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EVALUATION OF HEADSTART HAS AS ITS GENERAL CRITERION OF ‘ I 

EFFECTIVENESS THE RATIO BETWEEN COST AND BENEFIT. IF THE 
LATTER CAN BE DEMONSTRATED TO JUSTIFY THE FORMER, THE 
INTERVENTION PROJECT IS A GOOD INVESTMENT IN TERMS OF 
IMMEDIATE HELP FOR THE DISADVANTAGED PRESCHOOLER AND IN LONG 
RANGE BENEFITS FOR HIS FAMILY, COMMUNITY, AND SOCIETY. IN 
TERMS OF A MODEL, INPUT (POPULATION) PLUS OPERATIONS (PROGRAM 
ATTRIBUTES) MUST YIELD OUTPUT (CHANGES IN POPULATION 
ATTRIBUTES). IT IS DIFFICULT TO ASSESS AN OVERALL PROGRAM 
WHICH HAS SUCH A LARGE NUMBER OF VARIABLES. BUT HEADSTART 
GOALS CAN BE TRANSLATED INTO OPERATIONAL DIMENSIONS. THUS. 

THE PHYSICAL AND MENTAL HEALTH. SOCIAL ATTITUDES AND 
BEHAVIOR. AND COMMUNICATION SKILLS OF THE CHILDREN BECOME 
SOME OF THE OUTPUT VARIABLES WHICH CAN BE MEASURED BY PRE- 
AND POST-TESTING. METHODS OF ASSESSMENT OF THESE VARIABLES 
INCLUDE THE USE OF STANDARDIZED TESTS, NEW TESTS DEVELOPED 
SPECIFICALLY TO MEASURE DISADVANTAGED POPULATIONS. FILES 
ACCUMULATED AT HEADSTART CENTERS, AND CLASSROOM OBSERVATIONAL 
PROCEDURES. APPLIED RESEARCH WILL BE EMPHASIZED IN THE 13 
EVALUATION AND RESEARCH CENTERS THROUGHOUT THE COUNTRY WHICH 
WILL IMPLEMENT THE ONGOING EVALUATION. BOTH CROSS-SECTIONAL 

AND LONGITUDINAL EVALUATION DESIGNS WILL BE NEEDED TO MEASURE 1 

HEADSTART'S SHORT AND LONG RANGE EFFECTS. (MS) ■ 
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THE EVALUATION OF PROJECT HEAR START: 



A Conceptual Statement 

John W. McDavid, Director , 

Research and Evaluation 
Project Head Start 

THE BASIC CONCEPT OF PROJECT HEAD START 

Project Head Start was designed not merely as a preschool readiness 
program, but as a comprehensive intervention into the entire process of 
early childhood development. Many aspects of the child* s development 
are to.be served, with basic objectives including improvement of the 
child* s physical and mental health, emotional and social development, 
conceptual and verbal skills, self-confidence and aspirations, family 
relations, and attitudes toward society and social institutions. The 
program is oriented toward affecting the child as an individual most 
directly, but is secondarily intended to influence the family and 
community to which he belongs. Head Start is not a uniform action pro- 
gram dealing with a uniform population. It deals, in various sections 
of the nation and under various circumstances, with a variety of sub- 
populations differing in age, ethnic origin, family stability, degree 
of socio-cu.ltur£il impoverishment, physical health, or other dimensions. 
The nature of organized intervention provided by Head Start in the 
early development of these children may vary according to available 
physical and personnel resources, or educational philosophies of those 
wlio administer the program. Furthermore, Head Start is designed to pro- 
duce multidimensional long range change in the population that it serves, 
rather than a single intended immediate outcome. 

The Division of Research and Evaluation of Project Head Start is 
responsible to its larger administrative structure, the Community 




2 



Action Program of the Office of Economic Opportunity, to provide data 
reflecting the overall national effectiveness of the Head Start program. 
The general criterion of effectiveness is, in the final analysis, one of 

cost/benefit ratio. It must eventually be demonstrable that investments 

% 

of money, time, and effort in the Head Start concept do in fact yield 
benefits which justify that investment. 

Professionals who have worked within the ar^ a of early childhood 
development are Inclined to assume on faith that the Head Start concept 
is a sound one. However, the Division of Research and Evaluation is 
charged with the responsibility of generating hard empirical data to 
demonstrate that the Head Start concept of broad- scale intervention in 
the dejrelopmental process can interrupt the poverty cycle and optimize 
the integration of disadvantaged children into the mainstream of .our 
society. While the efficacy of intervention through preschool programs 
to facilitate the development of middle-class children has been demon- 
strated in a variety of ways, there has been little exploration of 
similar kinds of intervention with economically disadvantaged children 
and their families. Thus, the Division of Research and Evaluation is 
charged with the task of demonstrating the replication of what we know 
about middle-class children with a new kind of population, as well as 
with generating new ideas and innovations with respect to maximizing 
effective kinds of intervention with disadvantaged .children. 

While the direct .i.uipact of Project Head Start is upon the indivir 
dual child, as a national action program it is intended to produce a 
considerably broader impact upon the family, the educational system, 
and eventually the adult society in which these children, will partici- 
pate. In this sense, the immediate and short-range effects of Head 
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Start participation upon the child are merely intermediate instrumental 
steps toward the alleviation of a set of social problems which stem 
primarily from the fact of economic poverty* Consequently, certain of 
the primary objectives of Head Start cannot, by their very nature, be 
ascertained and evaluated until some years in the future* Yet a strict 
and literal cost/benefit criterion of effectiveness would require 
gauging these effects immediately., 

For these reasons, the basic national evaluation of Head Start 
cannot immediately execute a full and comprehensive evaluation of the 
program* s true effectiveness; however, it can assess certain specific 
shorter range impacts of the program upon the child and his family, and 
provide accompanying supportive argument and evidence that these shorter- 
range effects may in turn link to the intended longer-range goals of 
Head Start. 



A GENERAL MODEL FOR THE EVALUATION OF PROJECT HEAD START 



The basic concept of Head Start described above can be represented 
by a "system" model in which certain operations (the comprehensive pre- 
school Head Start program) are carried out on certain populations 
(preschool age children and their families) to produce certai.n outcomes 

. 

or changes (objectives or goals of the Head Start endeavor) • This 
model can be schematized: 

INPUT X OPERATIONS OUTPUT 

(population (program (changes in 

attributes) attributes) population 

attributes or 
emergence of 
new attributes) 

In essence, this model suggests that any outpu t variable may be related 
to the interaction of input variables and operations variables. The 

f 

! . ■ 
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model need not imply any particular kind of investigative design, although 
it does imply the need for iraltivariate exploration# A correlational 
approach derived from this model might explore multiple correlations of a 
set of input and operations measures with a set of output measures. Or an 
analysis of variance approach mxght be derxved to xnvestxgate the xnter~ 
action of input and operations classifications in determining output 
dependent variable measures. 

Basic Units of Analysis . The model defines, implicitly, two basic 
units of investigation in the predictor or independent variable cate- 
gories ( input and operations) ! the basxc unxt of analysxs for xnput xs 
the child ; that for operations is the class . 

(a) INPUT Va riables . The individual child is the primary basic unit 
for definition of input variables. However, Head Start* s conception of 
the child is a comprehensive one: it includes not only the child as a 

ff 

biological organism, but also as a member of a family, a community, or 
other social systems. His participation in such social systems is a part 
of the broader conception of him as an xndxvxdual. In thxs category are 
included physical attributes (e.g., sex, age, racial origin, physical 
health and make-up, etc.), socio-cultural attributes (e.g., economic 
level of his family, ethnic background, community background, linguistic 
environment, family structure and stability, etc.), attitudinal attributes 
(toward himself, his family, his school, his community, or other social 
institutions and individuals), and behavior al attributes (6«S*» aggressive 
ness, sociability, initial cognitive-intellectxial abilities, etc.) as they 
characterize the child prior to exposure to Head Start program operations. 



mmsmsmsm. 
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(b) OPERATIONS Variable s, The basic meaningful unit for assessing 
Head Start *s operation is the classroom unit . This is the social milieu 
in which the operations of a Head Start program are carried out, and is 
thus the minimum homogeneous unit of program operation. Although 
various classes x^ithin a particular Head Start Center (or set of centers) 
may be relatively homogeneous, it is nevertheless more appropriate to 
attempt to characterize program operations at the level of the class of 
15-20 children who comprise a meaningful group under the coordinating 
direction of one primarily responsible teacher. In this category of 
variables are included characteristics of the physical context in which 
the proijram operates (esg. ,' physical plant and materials, indoor and 
outside space and facilities, toys and educational materials, etc.), the 
adult personnel who implement the program (e.g., their personal and 
behavioral attributes, their attitudes, their qualifications and training, 
their exposure to and interaction with each child) , the peer-group of the 
class (e.g., their collective characteristics in terms of age, sex, or 
racial distribution, their average intellectual ability or average socio- 
economic status, their structure and function as a group, etc.), the 
services offered by the program (e.g., medical-dental diagnosis and 
treatment, psychological diagnosis and treatment, social services to the 
family, etc.), and the curricular patterns of the program (e.g., the 
formal characteristics of organized activity in the classroom, avail- 
ability and usage of ma* trials in direct training, both within and outside 
the classroom, specific modification of behavior through systematic 
modeling or reinforcement, etc.). 

(c) OUTPUT Variables . Although the basic unit for analysis of output 
dimensions x^ithin this model is the input unit of the individual child. 












the output impact of Head Start is likely to extend broadly beyond the 
individual child and his family. It is most pertinent to assess direct 
changes in the individual ch3.1d (both behavxoral xncludxng cognxtxvCj 
affective, and motivational dimensions, — as well as physical 
including medical health, nutrition, and the like, -- and social 
including the child's interactions vzithxn his family, his school, his 
peer-group, or his community. But evaluative assessment should extend 
further into the family (e.g., with respect to parent-child interaction, 
parental expectations and aspirations, modification of parental atti- 
tudes and behaviors not only toward the child but also toward the social 
institutions in which he participates, including his peer-group, his 
school, and his community). It may even extend further to evaluation of 
the impact of Head Start upon school systems and communities themselves 
as systems in which Head Start children now or eventually will partici- 



pate. 



The basic responsibility of the Division of Research and Evaluation 
for the overall evaluation of Head Start's effectiveness requires the 



accumulation of information about specific output variables which repre- 



sent translations of the stated goals of Head Start into o^a^a^tion^ 
dimensions. Consequently, the output dimensions evaluated must include 



assessment of such variables as the child's : 



1 . 

2 . 

3 . 



4 . 

5 . 

6 . 

7 . 

8 . 



physical health 

cognitive and intellective functions 

motivational characteristics (including particularly achievement 
motivation and goal-setting behavior with respect to learning 



and education) 
communication skills 
social behavior and peer interaction 

attitudes toward authority and adult-child interaction 



attitudes toward himself 

(self-esteem, self-confidence, and aspirations) 
attitudes and behavior with respect to his family, his school, 
his community, and social systems in which he narticinates. 
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Methods of Assessment c> No single method of assessment is suitable 
for all of the pertinent dimensions on which the Head Start program should . 
be evaluated. A variety of methodological approaches will be required, 
according to the nature of variables explored within any given aspect of 
the evaluation program. 

(a) Standardized Tests . Whenever possible, standardized tests should 
be used for evaluational assessment, both for econony and to facilitate 
integration of evaluative research findings with other literature on early 
childhood development. However, a standard instrument should not be used 
in the face of strong evidence of its inappropriateness for use with dis- 
advantaged children. 

(^) D evelopnient of New Tests and Procedures . The development of new 
tests and procedures for measuring pertinent dimensions should be an 
integral part of the overall evaluation effort of Project Head Start, 
especially because most existing tests have been designed for and standard- 
ized with middle-class populations of children and thus are likely to be 
less suitable for use with Head Start populations. New instruments may be 
required for the measurement of variables for which no standardized 
instrument currently exists. But it may be unwise to develop new instru- 
ments hurriedly on the assumption that existing instruments are inappro- 
priate for disadvantaged children without accompanying empirical investi- 
gation to determine the particular dimensions or qualities which make them 
inappropriate. 

(c) Files and Records . The administration of a Head Start program 
necessarily requires the accumulation of a considerable amount of file 
material on each child as well as upon the facilities and operation of the 
program and its personnel. These are important sources of data which should 
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be exploited fully, especially in. order to minimize excessive requests of 
Head Start personnel to respond to questionnaires and interviews. 

M) Observational Procedures . Despite their cost in time, effort, 
and expense of development, direct observational procedures may be 

A 

required to secure certain pertinent data. Since standardized observation 
is essentially a form of testing (i.e., sampling behavior), the develop- 
ment of new procedures in this category should be an integral part of the 
overall Research and Evaluation program, and should proceed in the same 
manner as the development of new test instruments. Procedures of this 
type are particularly pertinent to the assessment of program operations 
carried out in the Head Start class unit. 

EVALUATION DESIGNS: The multifaceted nature of Head Start's output 

goals requires more than one kind of evaluation design. Because Head 
Start's purpose is to bring about changes in attributes of the child and 
his family, these designs are basically of the "with/without" variety. In 
one type, a "pre/post" design, each child may serve as his own control, 
with output change measured by a comparison of some measure obtained prior 
to his exposure to Head Start and a parallel measure obtained afterward. 

In another type of design, measures on a child may be compared against 
measures on a matched counterpart who has not had such Head Start exposure 
(the control alter, in this sense, representing a "sans Head Start" 
measure). The second design may be somewhat more difficult to implement, 
since Head Start children and families are normally not randomly selected 
from the disadvantaged population, and are thus not necessarily directly 
comparable to those who do not participate in the program. The disad- 
vantaged population is often an invisible and inaccessible one, further 
complicating the attempt to secure baseline data for evaluation of Head 




start participants. 



(a) Cross»^sectional Designs . Certain kinds of useful information may 
be derived through cross-sectional assessments of Head Start. Designs 
which compare various subpopulations of Head Start children, or various 
treatment differences within Head Start programs, may be useful. Cross- 
sectional evaluation of the external impact of Head Start programs on the 
community, schools, or the public in general may supply important evalua- 
tive information. 

(b) Short-range Change . Additional useful information is needed about 
the immediate and short-term' impact of Head Start upon the child and his 
family. Ideally, "pre-Head Start" assessment should begin prior to the 
child's initial contact with Head Start, but this is practically impossible 
within the subject-as-his-own-control design in most cases. However, it 
might be conceivable to compare Head Start children with matched controls in 
a community where no Head Start has ever existed; or to test potential Head 
Start populations prior to the establishment of a Head Start center in the 
community. Both would be troublesome and costly to execute, but this kind 
of exploration may be warranted in order to illuminate "Hawthorne effects" 
in the evaluation of Head Start children. The very fact of registration and 
entry into a Head Start program may produce some change in children and their 
families, or may be systematically associated with certain "volunteer" 
characteristics of the Head Start population, or with selectivity in the 
location of Head Start centers. 

With increasing intervals of exposure to Head Start intervention, a 
variety of changes m the child and his family may occur . Since there 
presently exists little empirical basis for anticipating the speed with 
which particular change dimensions become apparent as a consequence of 
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Head Start intervention, a broad range of explorations of a variety of 
variables measured over a variety of intervals will be required to gain 



full knowledge of Head Start* s short-range impact. It is difficult to 
define a priori the points in time which best define "pre" and "post”, and 
a broad empirical effort must be undertaken to answer these questions. 

(c) Long-range Change . The ultimate effectiveness of Head Start as 



an interruption of the poverty cycle can be evaluated only with extended 
longitudinal approaches. The fact that Head Start has existed less than 
three years directly precludes this kind of investigation for the immedi- 
ate future. However, the current data-collection efforts of the Research 
and Evaluation program must be planned to optimize the utility of these 



data 




storage and eventual use in future long-range longitudinal 



studies. Provision has been made for a limited amount of data storage, 
but it is important that data collected in current research and evaluation 
not only be stored, but that the conditions of their collection be fully 
and accurately recorded, and that each item of data be fully identified 



with respect to its source. 

Provision must be made for longitudinal studies which follow Head 
Start children to assess their continued developmental progress. Attrition 
may be particularly great because of the charactenstxc mobxlxty of the 
(ji53^(jYantaged populatxon, but opportunxtxes to contxnue longxtudxnal 
studies may occur through cooperation with public school systems. 

In the absence of opportunity for direct longitudinal investigations 
in the immediate future, an alternative approximation of evaluation of long 
range effectiveness of Head Start may be developed through investigations 
Qj|i ^ 2_jYiGnsions of short-term change as xntermedxate consequences v/hxch are 
related to the longer-range objectives of Head Start. Changes in family 
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structure and parent~child interaction patterns may be particularly 
meaningful in terms of sustained modification of the child s developmental 
context beyond the duration of his Head Start experience* Changes in 
motivational, attitudinal, value, or interest patterns may be pertinent to 
long-range change, 

IMPLEMENTATION OF THE RESEARCH AND EVALUATION PROGRAM 

It is important that the concepts of Research and Evaluation not be 
widely divergent. The model for evaluation described in this statement is 
essentially a model for research. The very fact of the novelty of the 
Head Start program and the lack of prior direct knowledge of the natuie of . 
the disadvantaged lower-class population to which Head Start is addressed 
make necessary a certain amount of preliminary investigation before ade- 
quate evaluational data can be secured. Pertinent dimensions must be 
identified and instruments must be developed to permit their measurement. 
Previous research findings based on middle-class populations must suffer 
replication to ascertain their gener lizability to the disadvantaged 
populations. Descriptive data must be accumulated to permit full descrip- 
tion of Head Start*s population and program characteristics as they in 
fact exist at the level of actual implementation of stated ideals and 
goals. It would seem unwise to attempt a sharp delineation of the 
boundary between research and evaluation. 

However, it will be necessary to recognize that the research con- 
ducted within this program is by definition Q^PP . ll.^^d research. Such recog- 
nition need not connote a sharp dichotomy between "basic" and "applied" 
research. Applied research is simply that vzhich is guided by some attri- 
but6 of inuTiBdiate applicaboMity to a practical ncod# The financial 
support of Head Start’s research program requires that our research efforts 



be applicable to the task of assessing the effectiveness of Head Start as 
a social action program, but this does not necessarily imply that such 
applied research is expected to produce only certain "desirable" findings. 
The central office in Washington of the Division of Research and 
Evaluation functions as a coordinating agency so that the activities of 
individuals and agencies with whom it contracts can be integrated into an 
overall pattern which fulfills the need for evaluation of a national 
action program. 

The Research and Evaluation program is carried out through several 
means: 

(^) Regioiial Evaluation and Research Centers , A basic set of 13 
contracted University Centers for Head Start evaluation and research, 
appropriate to the inherent diversity of Head Start operations in various 
sections of the nation, conduct a continuing program of evaluation, using 
common procedures within an overall coordinated design. 

The independent research activities of the E & R Centers are generally 
planned to facilitate some degree of integration of findings, cross- 
validating replications of studies, and exchange of data among centers, 
Othen^ise, their research activities are conducted and administered in the 
same manner as independently contracted research investigations. 

The sampling design for 1967-68 is focused upon the individual class 

unit. Data are collected according to a general "pre/post" design, 

> 

Common instruments and procedures are used to evaluate dimensions defined 
by the general overall concept and objectives of Project Head Start, The 
Regional E 6e R Centers address their efforts primarily toward the internal 
evaluation of Head'' Start’ s operation, with limited excursion into family 
impact and community impact. Information for evaluating the latter is 
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secured through other sources, while the E & R Center evaluation is 
addressed to the individual child and classroom unite It might be possi-* 
ble to evaluate a non Head Start population using identical data-collection 
procedures to provide a general and nonspecific control for interpretation 
of the findings of the 13 E & R Centers, but it would be important m to 
consider this a genuine control in a precise sense, but merely as a non- 
specific point of reference for interpretation of the data from the 
national evaluation sample. Neither Head Start participants nor non- 
participants are randomly representative of a single population, and care 

» 

must be taken in interpreting comparisons of the two groups. 

(b) Individual Research and/or Demonstration Contracts. In areas 
extending beyond the function of the Regional Centers, the Head Start 
Evaluation and Research program is carried out through contracts for 
smaller packaged evaluative investigations. These contracts are awarded 
within the framework of an overall design, according to the merits of pro- 
posals in terms of their supplementation of the overall evaluation program. 
An announcement outlining the overall intent of the Division of Research 
and Evaluation is prepared for national distribution among professional 
researchers in pertinent areas. 

The politically sensitive nature of Head Start as a controversial 
national action program requires particular caution with respect to the 
issues of "informed consent" of subjects in research investigations and of 
"invasions of privacy" in the use of questionnaires and other procedures 
for collecting data about individuals and the storage of information about 
people in data banks. It is necessary for the Head Start Research and 
Evaluation office to remain especially sensitive to these issues and to 
protect both investigator and subjects. Unfortunately, certain categories 
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of data which are of genuine significance in research on the preschool 
child and his relationships with his family and social institutions may 
have to be sacrificed regardless of their relevance to the overall 
research and evaluation program. It is necessary that OEO-funded investi- 
gators secure, through the Head Start Research and Evaluation office, 
clearance for all questionnaires, tests, or procedures employed in the 
collection of data. 

CONTINUITY OF THE RESEARCH AND EVALUATION PROGRAM 

Continued implementation of the Head Start research and evaluation 
program is contingent upon the overall future of Head Start as an action 
program, but the Division of Research and Evaluation operates under the 
working assumption that Head Start will cpntinue and that its parallel 
research and evaluation efforts will continue. Provision should be made 
JEor accumulation and storage of fully-identified data for potential use 
in future longitudinal studies. Longitudinal studies following Head Start 
children through their elementary school years are appropriate, within the 
limitation of available funds which necessarily restricts the duration over 
which studies can be funded. Each succeeding year*s evaluation and 
research program should be formulated upon the results of the preceding 
year*s effort. 

Periodic integrations of research and evaluation findings are pre- 
pared from time to time by the central office professional staff, 
occasionally enlisting the collaboration of distinguished professional 
consultants according to the nature of emergent findings. 

As rapidly as accumulating information permits, research and evalu- 
ation efforts should move into a phase of testing specific hypotheses and . 
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expected relationships. For the present, such hypothesis-testing remains 
somewhat limited, since our reservoir of information about pertinent 
variables and methods for their assessment remains limited. As the program 
evolves, it will be possible to develop specific recommendations for maxi- 
mizing the effectiveness of Head Start, and the conduct of evaluative 
research to ascertain the merit of these recommendations. 

At this stage, it becomes increasingly important to provide for the 
establishment of experimental demonstration projects in which interventive 



practices which have been found (in laboratory studies or through the 
general national evaluation) to optimize attainment of Head Start's stated 



goals, can be evaluated carefully and demonstrated for adoption by other 



Head Start programs. 

The general framework described here, governs the general policy 



structure of Head Start's Research and Evaluation program, and guides 
decisions on fund allocations and continuing program planning for future 



years. 



